Priority-aware hierarchical communication traffic scheduling

ABSTRACT

Priority-aware hierarchical communication traffic scheduling methods and apparatus are disclosed. Communication traffic is scheduled by schedulers. Each scheduler supports a multiple priority scheduling function to schedule communication traffic having any of multiple priorities. The communication traffic scheduled by the schedulers is further scheduled based on the priorities of the communication traffic scheduled by each of the schedulers. Traffic shaping may also be integrated into a hierarchical scheduler. The priority-aware further scheduling and the optional shaping may be implemented, for example, using multiple Active lists, one per traffic priority, and a Pending list to which a scheduler can be added if it has communication traffic to schedule but is not allowed to schedule that traffic due to the shaping.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.11/976,459 filed on Oct. 24, 2007. The entire contents of the relatedpatent application are incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to communications and, in particular,to priority-aware scheduling of communication traffic.

BACKGROUND

Scheduling is done in multiple stages in a hierarchical scheduler.Leaf-level schedulers in a hierarchical scheduler might include somecombination of Strict Priority (SP) and Weighted Round Robin/DeficitRound Robin (WRR/DRR) schedulers. Higher levels of a scheduler mightinclude a combination of optional shapers followed by SP, Time DivisionMultiplexing (TDM) calendar, or DRR scheduling. None of these algorithmsused in a second or higher level of a scheduler hierarchy factors indynamically changing priority of the traffic scheduled by lower levelmembers.

Traditional DRR scheduling, for example, employs a single Active-statelist that is not priority-aware, and also does not provide for shaping.Although Modified DRR or (MDRR) scheduling might use one or more SPqueues and a DRR at the lowest priority level, the combination isscheduled in a fixed priority order. The SP queues and the DRR schedulerhave fixed priorities relative to each other, and only that fixedpriority order is applied in subsequent scheduling among the queues andthe DRR scheduler.

SUMMARY OF THE INVENTION

Thus, there remains a need for improved techniques for priority-awarehierarchical scheduling.

According to an aspect of the invention, there is provided an apparatuscomprising: a plurality of schedulers that schedule communicationtraffic, each scheduler supporting a multiple priority schedulingfunction to schedule communication traffic having any of a plurality ofpriorities; and a priority-aware scheduler, operatively coupled to theplurality of schedulers, that further schedules communication trafficfrom the plurality of schedulers based on priorities of communicationtraffic scheduled by each of the plurality of schedulers.

The priority-aware scheduler may further schedule traffic based onpriorities by maintaining a plurality of Active lists respectivelyassociated with the plurality of priorities. Each scheduler of theplurality of schedulers may schedule communication traffic to the Activelist associated with the priority of communication traffic it iscurrently scheduling.

In some embodiments, the priority-aware scheduler services the pluralityof Active lists in priority order according to a DRR schedulingalgorithm.

The apparatus may also include a shaper, operatively coupled between ascheduler of the plurality of schedulers and the priority-awarescheduler, that controls scheduling of communication traffic to thepriority-aware scheduler by the scheduler to which the shaper isoperatively coupled.

The shaper may allow the scheduler to which it is operatively coupled toschedule communication traffic to the priority-aware scheduler at up toa maximum rate. The scheduler can be added to a Pending list after theshaper has reached its maximum rate.

In some embodiments, the shaper manages the maximum rate by issuingtokens to the scheduler to which it is operatively coupled. Thescheduler schedules communication traffic to an Active list when it hascommunication traffic to schedule and it has sufficient shaper tokens toschedule the communication traffic, and is added to the Pending listwhen it has communication traffic to schedule but it does not havesufficient shaper tokens to schedule the communication traffic.

The shaper may be one of a plurality of shapers operatively coupledbetween respective schedulers of the plurality of schedulers and thepriority-aware scheduler, with each shaper controlling scheduling ofcommunication traffic to the priority-aware scheduler by the schedulerto which the shaper is operatively coupled. Each shaper may allow thescheduler to which it is operatively coupled to schedule communicationtraffic to the priority-aware scheduler at up to a respective maximumrate. A scheduler that is operatively coupled to a shaper can be addedto a Pending list after its shaper has reached its maximum rate.

A scheduler that is operatively coupled to a shaper may move from thePending list to an Active list when it has communication traffic toschedule and it is issued sufficient tokens to schedule thecommunication traffic by its shaper.

The plurality of shapers may include shapers having different respectivemaximum rates.

A method is also provided, and includes: scheduling communicationtraffic in a plurality of schedulers, each scheduler supporting amultiple priority scheduling function to schedule communication traffichaving any of a plurality of priorities; and further scheduling thecommunication traffic scheduled in the plurality of schedulers based onpriorities of communication traffic scheduled by each of the pluralityof schedulers.

The further scheduling may involve maintaining a plurality of Activelists respectively associated with the plurality of priorities, in whichcase the scheduling in the plurality of schedulers involves eachscheduler scheduling communication traffic to the Active list associatedwith the priority of communication traffic it is currently scheduling.

In some embodiments, the further scheduling involves servicing theplurality of Active lists in priority order according to a DRRscheduling algorithm.

The method may also include shaping the communication traffic that isscheduled by a scheduler of the plurality of schedulers prior to thefurther scheduling. Shaping may involve allowing communication trafficto be scheduled by the scheduler at up to a maximum rate, andmaintaining a Pending list to which the scheduler is added after themaximum rate is reached.

The allowing may involve managing the maximum rate by issuing tokens tothe scheduler, in which case the scheduling by the scheduler may involvescheduling communication traffic to an Active list when it hascommunication traffic to schedule and it has sufficient shaper tokens toschedule the communication traffic. The scheduler is added to thePending list when it has communication traffic to schedule but it doesnot have sufficient shaper tokens to schedule the communication traffic.

In some embodiments, scheduling involves moving a scheduler from thePending list to an Active list when it has communication traffic toschedule and it is issued sufficient shaper tokens to schedule thecommunication traffic.

Different respective maximum rates may be used for the shaping fordifferent schedulers.

According to another aspect of the invention, there is provided anapparatus comprising: an interface that enables the apparatus to receivecommunication traffic from a plurality of schedulers that support amultiple priority scheduling function to schedule communication traffichaving any of a plurality of priorities; and a priority-aware scheduler,operatively coupled to the interface, that further schedulescommunication traffic scheduled by the plurality of schedulers based onpriorities of communication traffic scheduled by each of the pluralityof schedulers.

The priority-aware scheduler may further schedule communication trafficby maintaining a plurality of Active lists, respectively associated witha plurality of communication traffic priorities, to which the pluralityof schedulers can schedule communication traffic.

In some embodiments, the priority-aware scheduler services the pluralityof Active lists in priority order according to a DRR schedulingalgorithm.

Other aspects and features of embodiments of the present invention willbecome apparent to those ordinarily skilled in the art upon review ofthe following description.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of embodiments of the invention will now be described ingreater detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of a DRR architecture.

FIG. 2 is a block diagram of an MDRR architecture.

FIG. 3 is a block diagram of a hierarchical scheduler according to anembodiment of the invention.

FIG. 4 is a flow diagram illustrating a method according to anembodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 is a block diagram of a DRR architecture. The DRR architecture 10shown in FIG. 1 includes traffic queues 12, 14, 16, and a DRR scheduler18. The queues 12, 14, 16 are serviced in a round robin fashion, withthe DRR scheduler 18 scheduling up to a quantum of traffic from eachqueue in each round. Any remainder of the allowed quantum for a queueafter the queue is serviced in a round, also referred to as a “deficit”,is carried forward to the next round. Further details of DRR schedulingmay be found, for example, in: M. Shreedhar and George Varghese,“Efficient Fair Queuing using Deficit Round Robin”, Proc. of ACM SIGCOMM'95, August 1995.

FIG. 2 is a block diagram of an MDRR architecture. The architecture 20includes a traffic queue 22 from which traffic is directly scheduled bythe SP scheduler 36, and three traffic queues 28, 30, 32 from whichtraffic is initially scheduled by a DRR scheduler 34 and then by the SPscheduler 36. This represents a traditional form of MDRR, which involvesSP scheduling among a strict priority queue and the output of a DRRscheduler. When a scheduling request is received by an MDRR scheduler,the SP queue 22 is scheduled. If the SP queue 22 is empty, then thequeue 28, 30, 32 picked by the DRR scheduler 34 is scheduled.

An overview of other example implementations of MDRR can be found, forexample, in: Luciano Lenzini, Enzo Mingozzi, and Giovanni Stea,“Bandwidth and Latency Analysis of Modified Deficit Round RobinScheduling Algorithms”, Valuetools '06, Pisa, Italy, 11-13 Oct. 2006.

According to an aspect of the present invention, a modified version of aDRR algorithm is used in hierarchical scheduling. Jitter of highpriority traffic in a multi-level DRR instance can thereby be reduced.

As described in further detail herein, embodiments of the presentinvention may provide for priority-aware scheduling of communicationtraffic among lower-level schedulers, taking into account the prioritiesof traffic that is currently scheduled by those lower-level schedulers.Where each scheduler supports a multiple priority scheduling function toschedule traffic having any of a number of different priorities, thepriority of traffic being scheduled by a lower-level scheduler at anytime may dynamically change. A higher-level scheduler implementing anembodiment of the invention schedules traffic from its lower-levelschedulers in accordance with such dynamic priorities.

Consider a case where a leaf-level scheduler is an MDRR and the nexthigher-level scheduler schedules across multiple member MDRRs. In thisexample, one of the member MDRRs (MDRR1) can schedule priority 1 trafficand at the same time another member MDRR (MDRR3) can schedule priority 3traffic. However, if a standard scheduler is employed at the next higherlevel to schedule among multiple of those leaf-level MDRRs, it will notpay attention to the priority levels of the traffic scheduled byindividual members.

For instance, an SP scheduler will schedule the member MDRRs in a fixedand static priority order that is configured and not changed, a TDMcalendar will not pay attention to priority, and a DRR will schedulefrom a single Active list based only on deficits and packet lengths,without differentiating the two member MDRRs based on priority. Becausesuch standard higher-level scheduler algorithms are not aware of dynamicchanges in priority, any of those may introduce jitter, especially whenthe number of members at the leaf level is high (>32).

An algorithm according to an aspect of the present invention, however,will pay attention to dynamic changes in priority of the trafficscheduled by its members, and puts them in an appropriate one of anumber of lists in some embodiments. As a result, jitter may be reducedat all priority levels. In the above example, MDRR1 might be in Activelist #1, with MDRR3 being in Active list #3. Assuming that Active list#1 is associated with higher priority traffic than Active list #3,Active list #1 will be serviced first, followed by Active list #3. Sincethe decision dynamically depends on the priority level of trafficselected by a leaf member, MDRR1 may move to Active list #2 if the nextpacket it schedules has priority 2, for example.

Some embodiments also integrate one or more shapers into a hierarchicalscheduler.

FIG. 3 is a block diagram of a hierarchical scheduler according to anembodiment of the invention. The hierarchical scheduler 40 includesmember schedulers 48, 58, which schedule communication traffic fromqueues 42/44/46, 52/54/56, optional shapers 64, 66 operatively coupledbetween the schedulers 48, 58 and the higher-level priority-awarescheduler 62, and a memory 68 and a controller 69 that are operativelycoupled to each other and to all of the other scheduler components. Thenumeral 60 in FIG. 3 is intended to generally denote an interface thatenables the priority-aware scheduler 62 to further schedule traffic thathas already been scheduled by the schedulers 48, 58.

It should be appreciated that the scheduler 40 of FIG. 3 is intendedsolely for illustrative purposes, and that the present invention is inno way limited to the particular example embodiment explicitly shown inthe drawing and described herein. For example, a hierarchical schedulercould include more than the two member schedulers 48, 58, and/or morethan two levels. A hierarchical scheduler according to an embodiment ofthe invention could be used at any of those levels, including the memberMDRR level shown in FIG. 3. Also, as noted above, the shapers 64, 66 areoptional, and embodiments of the invention can be implemented withoutshapers, or with shapers only for some of the member schedulers 48, 58in a hierarchical scheduler. Shapers might be provided for alllower-level schedulers, but selectively enabled for all or only certainones of the schedulers.

The memory-based implementation shown in FIG. 3 is also intended to beillustrative. Each of the schedulers 48, 58, 62, as well as each shaper64, 66, may maintain its own configuration and state information inregisters and/or memories. In a single-memory implementation, there isone piece of physical memory that stores information an entry for eachscheduler and shaper. Each component of a hierarchical scheduler mayinstead store its configuration and state information, and/or possiblyother information, in respective separate registers or memories. Amemory-based implementation might be especially appropriate when thenumber of schedulers is relatively high, such as greater than 32,because memories tend to have a smaller die area and be easier forphysical design than many discrete registers. However, having registersor memories is simply an implementation choice, and the presentinvention is in no way restricted to either of these types ofimplementations.

More generally, other embodiments may include further, fewer, and/ordifferent components interconnected in a similar or different mannerthan shown.

Those skilled in the art will appreciate that the queues 42, 44, 46, 52,54, 56 may be implemented in any of various forms, using a variety ofmemory devices. Solid state memory devices are commonly used for thispurpose, although other types of memory devices using movable or evenremovable storage media could also or instead be used. These queuesstore communication traffic that may be received from any of a number oftypes of traffic sources, including user communication devices, forexample, while that traffic is waiting to be scheduled.

The memory 68 may also include one or more memory devices of any ofvarious types. As discussed in further detail below, the memory 68 maybe used to store Active lists and a Pending list that are used in someembodiments.

Software, hardware, firmware, or combinations thereof may be used toimplement the schedulers 48, 58, the shapers 64, 66, the priority-awarescheduler 62, and the controller 69. Any or all of microprocessors forexecuting software stored on a computer-readable medium, such as in thememory 68, Application Specific Integrated Circuits (ASICs),Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays(FPGAs) are examples of devices that may be suitable for implementingthese components. Given such a broad range of possible implementations,these components are described below primarily in terms of theirfunctions. Based on these functional descriptions, a person skilled inthe art would be enabled to implement embodiments of the invention inany of various ways.

Considering the actual interconnections between the components shown inFIG. 3, these may, at least to some extent, be implementation-dependent.Physical connections such as traces on an electronic circuit card, othertypes of wired connections, and wireless connections, as well as logicalconnections between software components through shared memory areas orregisters, represent examples of operative couplings that may beprovided in embodiments of the invention.

In operation, the schedulers 48, 58 schedule communication traffic fromthe queues 42/44/46, 52/54/56, respectively. Each scheduler 48, 58supports a multiple priority scheduling function to schedulecommunication traffic having any of a number of priorities. Although themember schedulers 48, 58 are MDRR schedulers in the example shown, otherembodiments of the invention may be implemented in conjunction withother types of member schedulers instead of or in addition to MDRRschedulers.

The priority-aware scheduler 62 is operatively coupled to the pluralityof schedulers, through the shapers 64, 66 in the example shown, andschedules communication traffic from the plurality of schedulers basedon the priorities of communication traffic scheduled by each of theschedulers 48, 58.

In one embodiment, the priority-aware scheduler 62 schedules traffic bymaintaining multiple Active lists, illustratively as lists in the memory68. The Active lists are respectively associated with different trafficpriorities. Each member scheduler 48, 58 schedules communication trafficto the Active list associated with the priority of communication trafficit is currently scheduling.

The effect of an Active list itself could potentially be implemented ina number of ways. For example, an Active list might include anidentifier of each member scheduler 48, 58 that has scheduled traffichaving the priority associated with that Active list. In this case, eachActive list is a list of schedulers that have scheduled traffic of acertain priority. An Active list might also or instead be used to storean indication of the particular queue 42/44/46, 52/54/56 from whichtraffic has been scheduled by the member schedulers 48, 58, and possiblyan indication of the priority of the scheduled traffic. Another optionwould be to store in each Active list the actual scheduled traffic, suchas a packet, that the scheduler has scheduled.

The present invention is in no way limited to any particular scheme formanaging Active lists, and references herein to scheduling traffic to anActive list is intended to convey all of the above options, as well asothers that may be or become apparent to those skilled in the art.

It should also be noted that a member scheduler 48, 58, and thepriority-aware scheduler 62 as well, need not necessarily output anytraffic by themselves; they need not even hold any actual traffic orpacket data. A scheduler might be considered a sort of manager thatdecides which queue's traffic is to either proceed to next levelscheduling or be output. A scheduler itself does not need to taketemporary “ownership” of the traffic it schedules, even though it mayhave integrated intelligence and memory to handle packet data.

According to one embodiment, the priority-aware scheduler 62 servicesthe Active lists in priority order according to a DRR schedulingalgorithm, although other algorithms could potentially be used.

In the example hierarchical scheduler 40, the shapers 64, 66 arerespectively operatively coupled between the schedulers 48, 58 and thepriority-aware scheduler 62.

The shapers 64, 66 are effectively gatekeepers that may allow or denyscheduling of traffic by a member scheduler 48, 58 to the priority-awarescheduler 62 pass or denying it. It is not outputting any traffic. Eachshaper 64, 66 controls the scheduling of traffic by its scheduler 48, 58to the priority-aware scheduler 62. The shapers 64, 66 allowcommunication traffic to be scheduled to the priority-aware scheduler 62by the member schedulers 48, 58 at up to a maximum rate, in order toprovide desired traffic patterns. The rates for the shapers 64, 66 maybe the same or different.

More generally, a set of shapers may include shapers that have the samerate and/or shapers that have different rates. To be clear, the samehierarchical scheduler might include some shapers that have the samerate and others that have different rates, or all of the shapers mayhave different rates. Any combination of common and different rates maybe provided, for example, by configuring the shapers with desiredshaping rates or patterns.

As noted above, the priority-aware scheduler 62 may managepriority-based scheduling using Active lists, with one Active list pertraffic priority level. A Pending list may also be maintained by thepriority-aware scheduler 62 to enable the shapers 64, 66 to manage theirmaximum rates. However, in the case of a Pending list, a single listcould be maintained for all member schedulers that have a shaper.

A scheduler 48, 58 that is operatively coupled to a shaper 64, 66 canthen make use of the Pending list when its shaper has reached itsmaximum rate. The Pending list, like the Active lists, might storescheduler identifiers, queue identifiers and possibly traffic priorityindications, and/or scheduled communication traffic.

The Pending list could instead be a Pending state, with 1 bit perscheduler 48, 58 indicating whether each respective scheduler is in aPending state waiting for shaper token updates, for example. Shapertokens are described in further detail below.

As noted above, Active lists may be implemented as link lists withqueues/schedulers as members. In one embodiment, the members areserviced in order, head-to-tail. A member might stay at the head of thelist as long as it has a deficit as defined in a DRR algorithm used bythe priority-aware scheduler 62. Pending list functionality couldeffectively be incorporated into the Active lists and a scheduler state.A member in an Active list might have a Pending state bit=0, whereas thesame member will have Pending state bit=1 when it runs out of shapertokens. Hence, in an implementation where there is no Pending list inthe form of an actual list but there is Pending list functionalitysupported by a Pending state bit, that bit could be part of everyscheduler state, with the value of the bit determining whether thescheduler is in an active or a pending state.

Another option for the Pending list would be to store entries in theform of {q,p}48, {q,p}58, where {q,p}m is a tuple identifying the winnerqueue (q) of scheduler m with priority p. The entry for scheduler m canbe updated every time a queue q with priority p higher than {q,p}m comesin, even when the scheduler m is waiting in the Pending list. Thismechanism could be used to ensure that higher priority traffic isscheduled before lower priority traffic when shaper tokens arereplenished, for example.

A similar mechanism could be used to allow lower-level schedulers tomove between Active lists before currently scheduled communicationtraffic has been scheduled from an Active list by a higher-levelpriority-aware scheduler. For example, if the priority p ofcommunication traffic changes for a scheduler that is in Active list mwhich is not for p, the scheduler can be removed from the Active list mand added to the tail of a different Active list n assigned for priorityp.

Shapers such as 64, 66 may use tokens to manage their maximum rates.Shaper tokens representing a total amount of traffic that a shaper canpermit to be scheduled within a certain period of time might berefreshed periodically. A shaper 64, 66 may issue tokens to itsscheduler 48, 58 to allow the scheduler to schedule blocks of traffic.For example, in some embodiments, the member schedulers 48, 58 schedulevariable length packets of traffic, and the shapers 64, 66 manage tokensthat each allow a certain number of bytes of traffic to be scheduled. Inthis case, the scheduler 48 is able to schedule a packet of traffic toan Active list when the length of the packet is less than or equal tothe total number of bytes corresponding to the number of tokens it hasbeen issued by its shaper 64. A scheduler 48, 58 that has traffic toschedule but does not have enough shaper tokens to schedule at least onepacket or block of traffic is added to the Pending list.

When tokens for a shaper 64, 66 are replenished, that shaper can againallow it scheduler 48, 58 to schedule traffic to the priority-awarescheduler 62. In this case, a scheduler 48, 58 that has scheduledcommunication traffic to the Pending list and is issued sufficienttokens by its shaper 64, 66 is “moved” from the Pending list to the endof an Active list since it is now permitted to schedule traffic again.The Active list that is chosen is the one that corresponds to thepriority of the traffic that the member scheduler 48, 58 is ready tosend at the time when its shaper tokens become positive.

The mechanism by which a scheduler 48, 58 moves from the Pending list toan Active list may be implemented in any of various ways. For example, ashaper 64, 66 may issue a token to, or otherwise alert its scheduler 48,58, when its tokens are replenished. The scheduler 48, 58 can thenremove itself from the Pending list and add itself to the appropriateActive list. The shaper 64, 66 itself could potentially move itsscheduler 48, 58 from the Pending list to an Active list.

In some embodiments, the controller 69 controls shaper alerts, updates,movement of schedulers in and out of the Active and Pending lists, andpossibly other functions. The controller 69, although shown in FIG. 3 asa single component, can be broken down into a group of smaller anddistributed controllers to control any or all of these functions in adistributed but co-ordinated fashion.

As shown near the interface 60, the priority-aware scheduler 62 mayreceive from the schedulers 48, 58, through the shapers 64, 66, anindication of a queue from which traffic has been scheduled, as well asthe priority of the scheduled traffic. The priority-aware scheduler 62can then further schedule the traffic based on priorities, which asnoted above may dynamically change, in that the lower-level schedulers48, 58 are not restricted to any particular traffic priority.

An output of the priority-aware scheduler 62 is similarly shown as anidentifier of a queue. In other embodiments, actual scheduled traffic,in the form of packets or other traffic blocks, could be transferredbetween the schedulers 48, 58 and the priority-aware scheduler 62,and/or output by the priority-aware scheduler.

In schedulers with more levels of hierarchy, multiple instances of thepriority-aware scheduler 62, or even an entire hierarchical scheduler40, may feed to the next level, which can similarly include apriority-aware scheduler. In that case, the priority-aware scheduler 62,like its member schedulers 48, 58, output {q, p} for use by the nextlevel. Hence, q is not necessarily the only output of the priority-awarescheduler 62. Traffic priority p may also be provided as an output ifthere is a next level of scheduling that has a priority-aware scheduler.Otherwise, if the level after the priority-aware scheduler 62 is notpriority aware, the priority-aware scheduler 62 may simply output q, asshown.

Thus, as will be apparent from the foregoing, each member scheduler 48,58 may select one of its traffic queues, and the priority-awarescheduler 62 effectively aggregates the decisions of the memberschedulers. The outputs of the member schedulers are optionally shaped.

If the higher-level scheduler were a standard DRR instead of thepriority-aware scheduler 62, it would select traffic from one of themember schedulers 48, 58 based on a configured quantum, withoutprioritizing between the member schedulers. If there are many memberschedulers, say 128, a color-blind aggregating DRR can result in highjitter of high priority traffic.

On the other hand, a priority-aware scheduler 62 might use multiplepriority-based Active lists instead of just one Active list as instandard DRR. A member scheduler 48, 58 in any Active list of theaggregating higher-level scheduler 62 can move to another Active list iftraffic having a different priority is available in any its queues. Amember scheduler 48, 58 can also move to the Pending list when it doesnot have any credits or tokens from the optional shaper that is shapingits output, and from the Pending list to an Active list whencredits/tokens again become available.

Embodiments of the invention have been described above primarily in thecontext of an apparatus. FIG. 4 is a flow diagram illustrating a methodaccording to an embodiment of the invention. The method 70 includeslower-level scheduling of communication traffic at 72. Each lower-levelscheduler supports a multiple priority scheduling function to schedulecommunication traffic having any of a number of priorities. Trafficscheduled by the lower-level schedulers is optionally shaped at 74, andfurther scheduling is performed at 76 based on the priorities of thetraffic scheduled by the lower-level schedulers.

The method 70 represents an illustrative embodiment of the invention.Other embodiments may include further, fewer, or different operationsperformed in a similar or different order. The operations shown in FIG.4 may also be performed in various ways. For example, a Pending listand/or shaper tokens may be used in shaping traffic at 74, and Activelists may be used for higher-level scheduling and shaping in someembodiments, as described above with reference to FIG. 3.

The method 70 may be recursive in some embodiments, with the operationat 76 effectively being replaced by the operations a 72, 74, and 76. Arecursive method is used where there are more levels of priority-awarescheduling. For example, with reference again to FIG. 3 for clarity,Voice, Video, Data queues of two DSL subscribers might be at the leaflevel similar to 42/44/46, 52/54/56. Those queues will be managed by thesubscriber level MDRR schedulers 48, 58, illustratively one persubscriber. Multiple subscriber MDRR schedulers 48, 58 may be aggregatedin a concentrator placed in a neighbourhood, with the concentrator levelcorresponding to the priority-aware scheduler 62. Several of theseneighborhood concentrators (aggregators) may be aggregated again at theCentral Office in a single piece of equipment, and that levelcorresponds to another priority-aware scheduler (not shown in FIG. 3),after the priority-aware scheduler 62. The Central Office equipment mayalso perform other functions such as Traffic Management for all levels,centrally, before sending the traffic downstream through theneighborhood concentrator.

Further variations may be or become apparent to those skilled in theart.

From the foregoing, it will be appreciated that embodiments of theinvention provide a scheduler algorithm suitable for second or higherlevel schedulers. Jitter of high priority traffic can be reduced in someembodiments by using a modified version of standard DRR. Modificationsmay include any or all of: multiple Active lists, including one list perpriority level; a new concept of a Pending list to integrate an optionalshaper into the algorithm for shaping the output of a lower-level memberscheduler; and a new DRR-like algorithm to move a lower-level memberscheduler from the Pending list to an Active list, and from one of theActive lists to another.

The higher-level scheduling algorithm includes the following behavioursin some embodiments: a scheduler that has no traffic to schedule is innone of the lists; a scheduler that has traffic scheduled at a certainpriority level (and also has sufficient shaper tokens in someembodiments) is a member of the Active list at that level; a schedulerthat has traffic scheduled and its shaper enabled but does not havesufficient tokens, is in the Pending list; a scheduler moves from thePending list to the Active list when sufficient shaper tokens becomeavailable; and a scheduler moves from one Active list to another Activelist when it has traffic scheduled and sufficient shaper tokens but itsscheduled traffic has changed priority.

What has been described is merely illustrative of the application ofprinciples of embodiments of the invention. Other arrangements andmethods can be implemented by those skilled in the art without departingfrom the scope of the present invention.

For example, embodiments of the invention may be applied to any ofvarious technologies, including but in no way limited to Gigabit-capablePassive Optical Network (GPON) products, WiMax Base Stations, WiFi LocalArea Network (LAN) controllers, Ethernet switches and routers, etc.

In addition, although described primarily in the context of methods andsystems, other implementations of the invention are also contemplated,as instructions stored on a computer-readable medium for instance.

1. An apparatus comprising: a plurality of schedulers that schedulecommunication traffic, each scheduler supporting a multiple priorityscheduling function to schedule communication traffic having any of aplurality of priorities; and a priority-aware scheduler, operativelycoupled to the plurality of schedulers, that further schedulescommunication traffic from the plurality of schedulers based onpriorities of communication traffic scheduled by each of the pluralityof schedulers.
 2. The apparatus of claim 1, wherein the priority-awarescheduler further schedules traffic based on priorities by maintaining aplurality of Active lists respectively associated with the plurality ofpriorities, and wherein each scheduler of the plurality of schedulersschedules communication traffic to the Active list associated with thepriority of communication traffic it is currently scheduling.
 3. Theapparatus of claim 1, wherein the priority-aware scheduler services theplurality of Active lists in priority order according to a Deficit RoundRobin (DRR) scheduling algorithm.
 4. The apparatus of claim 1, furthercomprising: a shaper, operatively coupled between a scheduler of theplurality of schedulers and the priority-aware scheduler, that controlsscheduling of communication traffic to the priority-aware scheduler bythe scheduler to which the shaper is operatively coupled.
 5. Theapparatus of claim 4, wherein the shaper allows the scheduler to whichit is operatively coupled to schedule communication traffic to thepriority-aware scheduler at up to a maximum rate, and wherein thescheduler is added to a Pending list after the shaper has reached itsmaximum rate.
 6. The apparatus of claim 2, further comprising: a shaper,operatively coupled between a scheduler of the plurality of schedulersand the priority-aware scheduler, that controls scheduling ofcommunication traffic to the priority-aware scheduler by the schedulerto which the shaper is operatively coupled, wherein the shaper allowsthe scheduler to which it is operatively coupled to schedulecommunication traffic to the priority-aware scheduler at up to a maximumrate, and wherein the scheduler is added to a Pending list after theshaper has reached its maximum rate.
 7. The apparatus of claim 6,wherein the shaper manages the maximum rate by issuing tokens to thescheduler to which it is operatively coupled, and wherein the schedulerschedules communication traffic to an Active list when it hascommunication traffic to schedule and it has sufficient shaper tokens toschedule the communication traffic, and wherein the scheduler is addedto the Pending list when it has communication traffic to schedule but itdoes not have sufficient shaper tokens to schedule the communicationtraffic.
 8. The apparatus of claim 6, wherein the shaper comprises oneof a plurality of shapers operatively coupled between respectiveschedulers of the plurality of schedulers and the priority-awarescheduler, each shaper controlling scheduling of communication trafficto the priority-aware scheduler by the scheduler to which the shaper isoperatively coupled, wherein each shaper allows the scheduler to whichit is operatively coupled to schedule communication traffic to thepriority-aware scheduler at up to a respective maximum rate, and whereineach scheduler that is operatively coupled to a shaper is added to aPending list after its shaper has reached its maximum rate.
 9. Theapparatus of claim 8, wherein a scheduler that is operatively coupled toa shaper moves from the Pending list to an Active list when it hascommunication traffic to schedule and it is issued sufficient tokens toschedule the communication traffic by its shaper.
 10. The apparatus ofclaim 8, wherein the plurality of shapers comprises shapers havingdifferent respective maximum rates.
 11. A method comprising: schedulingcommunication traffic in a plurality of schedulers, each schedulersupporting a multiple priority scheduling function to schedulecommunication traffic having any of a plurality of priorities; andfurther scheduling the communication traffic scheduled in the pluralityof schedulers based on priorities of communication traffic scheduled byeach of the plurality of schedulers.
 12. The method of claim 11, whereinthe further scheduling comprises maintaining a plurality of Active listsrespectively associated with the plurality of priorities, and whereinthe scheduling in the plurality of schedulers comprises each schedulerscheduling communication traffic to the Active list associated with thepriority of communication traffic it is currently scheduling.
 13. Themethod of claim 11, wherein the further scheduling comprises servicingthe plurality of Active lists in priority order according to a DeficitRound Robin (DRR) scheduling algorithm.
 14. The method of claim 11,further comprising: shaping the communication traffic that is scheduledby a scheduler of the plurality of schedulers prior to the furtherscheduling.
 15. The method of claim 14, wherein the shaping comprises:allowing communication traffic to be scheduled by the scheduler at up toa maximum rate; and maintaining a Pending list to which the scheduler isadded after the maximum rate is reached.
 16. The method of claim 12,further comprising: shaping the communication traffic that is scheduledby a scheduler of the plurality of schedulers prior to the furtherscheduling by allowing the scheduler to schedule communication trafficat up to a maximum rate, and maintaining a Pending list to which thescheduler is added after the maximum rate is reached.
 17. The method ofclaim 16, wherein the allowing comprises managing the maximum rate byissuing tokens to the scheduler, wherein the scheduling by the schedulercomprises scheduling communication traffic to an Active list when it hascommunication traffic to schedule and it has sufficient shaper tokens toschedule the communication traffic, and wherein the scheduler is addedto the Pending list when it has communication traffic to schedule but itdoes not have sufficient shaper tokens to schedule the communicationtraffic.
 18. The method of claim 16, wherein the shaping comprisesshaping the communication traffic that is scheduled by respectiveschedulers of the plurality of schedulers prior to the furtherscheduling by: allowing communication traffic to be scheduled by therespective schedulers at up to a respective maximum rate; andmaintaining a Pending list to which each scheduler of the respectiveschedulers is added after the respective maximum rate of the shaping isreached.
 19. The method of claim 18, wherein the scheduling comprisesmoving a scheduler from the Pending list to an Active list when it hascommunication traffic to schedule and it is issued sufficient shapertokens to schedule the communication traffic.
 20. The method of claim18, wherein the allowing comprises allowing communication traffic to bescheduled by the respective schedulers at up to different respectivemaximum rates.
 21. An apparatus comprising: an interface that enablesthe apparatus to receive communication traffic from a plurality ofschedulers that support a multiple priority scheduling function toschedule communication traffic having any of a plurality of priorities;and a priority-aware scheduler, operatively coupled to the interface,that further schedules communication traffic scheduled by the pluralityof schedulers based on priorities of communication traffic scheduled byeach of the plurality of schedulers.
 22. The apparatus of claim 21,wherein the priority-aware scheduler further schedules communicationtraffic by maintaining a plurality of Active lists, respectivelyassociated with a plurality of communication traffic priorities, towhich the plurality of schedulers can schedule communication traffic.23. The apparatus of claim 21, wherein the priority-aware schedulerservices the plurality of Active lists in priority order according to aDeficit Round Robin (DRR) scheduling algorithm.